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AN APPARATUS AND METHOD FOR DETERMINING A LEVEL. QSk 
CONFIDENCE IN A RISK ASSESSMENT AND/OR DETERMINING A LEVEL 



5 FIELD OF THE INVENTION 

The present invention relates to an apparatus and 
method for assisting a user in determining a level of 
confidence in a risk assessment and/or assisting the user 
10 in determining a level of acceptability of an event 
occurring . 

BACKGROUND OF THE INVENTION 

15 Risk is something which we all encounter in our 

everyday lives. For example, we risk being involved in an 
accident whenever we travel in a motor vehicle. Generally 
speaking, risk can be defined as the likelihood of some 
event occurring by the impact or consequences of that 

20 event; that is, risk =* likelihood * impact. 

A great deal of research has been undertaken in order 
to develop techniques for determining risk. Some of the 
more common techniques include: statistical analysis; 

2 5 forward simulations; mathematical modelling; and judgement 

or guessing. The problem with existing techniques for 
determining risk is that they do not take into account the 
validity of the information or process used to derive the 
risk. For example, where risk is determined using 

3 0 statistical analysis (of sampled data) the validity of the 

risk calculation (that is, the level of confidence a 
person can have in the risk calculation) will depend to 
some degree on the number of data samples used in the 
statistical analysis. Where only a few samples are taken 
35 the validity of the risk would be lower than if the risk 
were derived using a large number of samples. 
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Unfortunately, existing methods do not provide any 
indication as to the validity of a risk calculation. 
Consequently, a person using the risk calculation is 
unable to determine just how much confidence he/she can 
5 have in the calculated risk. This can be detrimental if, 
for example, a person has too much confidence in a risk 
calculation which is based on statistical analysis using 
only a few data samples (a low validity) . 

10 Existing techniques for determining risk assessment 

also do not give any indication as to the level of 
acceptability of the event occurring. 

SUMMARY OF THE INVENTION 

15 

According to a first aspect of the present invention 

there is provided an apparatus for assisting a user in 

determining a level of confidence in a risk assessment, 

the apparatus comprising: 
20 input means for receiving at least one of a set of 

factors each of which can affect the level of confidence 

in the risk assessment; 

storage means for storing the set of factors and a 

set of weighting indicators, wherein each factor within 
25 the set of factors is associated with a weighting 

indicator from the set of weighting indicators; 

processing means for processing the at least one of 

the set of factors entered into the input means and 

processing the set of factors stored in the storage means 
30 in order to identify a factor in the set of factors which 

corresponds to the at least one factor entered into the 

input means; and 

output means for outputting the set of weighting 

indicators and the weighting indicator associated with the 
35 factor identified by the processing means. 

Thus, the information output by the output means can 
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assist the user in determining the level of acceptance in 
the risk assessment. The set of weighting factors provide 
a context against which the weighting indicator associated 
with the factor identified by the processing means can be 
5 assessed. 

Preferably, the set of factors comprises: information 
about a technique used to obtain the risk assessment; the 
extent to which the technique is used to obtain the risk 
10 assessment; a level of acceptance of the technique and 

extent to which the technique is used to obtain the risk 
assessment; and information about the environment of the 
risk assessment. 

15 Preferably, the set of weighting indicators and the 

set of factors are arranged as a matrix, wherein the set 
of weighting indicators are entries in the matrix and the 
set of factors provide an index to the entries. 

20 Thus, by acting as an index to the entries, each of 

the set of factors is associated with a weighting 
indicator . 

Preferably, each of the set of factors providing the 
25 index to the entries in the matrix are divided into sub- 
factors . 

Preferably, the set of factors providing the index to 
the entries in the matrix comprises: the information about 

30 the environment of the risk assessment; information about 
techniques used to perform the risk assessment; an extent 
to which the technique is used to obtain the risk 
assessment; and a level of acceptance of the technique the 
extent to which the technique is used to obtain the risk 

35 assessment. 

Preferably, the stab- factors for the environment of 
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the risk assessment comprises: simple; low; moderate; high 
moderate; and complex. 

Preferably, the sub- factors for the extent to which 
5 the technique is used comprises: basic; moderate; and 
extensive . 



Preferably, the input means allows the user to enter 
and/or change the set of factors and the set of weighting 
1 0 indicators . 



Preferably, the input means is capable of storing in 
the storage means the set of factors and the set of 
weighting indicators entered into the input means. 

Preferably, the input means comprises a graphical 
user interface. 



Preferably, the storage means comprises a computer 
20 storage medium. 

Preferably, the computer storage medium comprises a 
database. 



25 Preferably, the processing means comprises a suitably 

configured computer. 

Preferably , the output means comprises a graphical 
user interface. 

30 

Alternatively, other forms of computer presentation 
devices can be used as the output means. For example, the 
output means could be a printer. 



35 According to a second aspect of the present invention 

there is provided a method for assisting a user in 
determining a level of confidence in a risk assessment, 
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the method comprises the steps of: 

entering into an input means at least one of a set of 
factors each of which can affect the level of confidence 
in the risk assessment; 
5 storing in a storage medium the set of factors and a 

set of weighting indicators , wherein each factor within 
the set of factors is associated with a weighting 
indicator from the set of weighting indicators; 

processing the at least one of the set of factors 
10 entered into the input means and processing the set of 

factors stored in the storage means in order to identify a 
factor in the set of factors which corresponds to the at 
least one factor entered into the input means; and 

outputting the set of weighting indicators and the 
15 weighting indicator associated with the factor identified 
by the processing step. 

Thus, the information output by the output means can 
assist the user in determining the level of acceptance in 
20 the risk assessment. The set of weighting factors provides 
a context against which the weighting indicator associated 
with the factor identified by the processing means can be 
assessed. 

25 Preferably , the method further comprises the step of 

arranging the set of factors as a matrix, wherein the set 
of weighting indicators are entries in the matrix and the 
set of factors provide an index to the entries. 

30 Thus, by acting as an index to the entries, each of 

the set of factors is associated with a weighting 
indicator. 



35 



Preferably, the method further comprises the step of 
entering and/or changing the set of factors and the set of 
weighting indicators entered into the input means. 
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According to a third aspect of the present invention, 
there is provided an apparatus for assisting a user in 
determining a level of acceptability of an event 
occurring, the apparatus including: 
5 input means for receiving a likelihood value which 

represents a likelihood that the event will occur, and a 
confidence value which represents a level of confidence 
that the user has in the likelihood value; 

storage means for storing a plurality of records each 
10 of which includes an indicator representing a level of 
acceptability of the event occurring, a range of 
likelihood values, and a range of confidence values; 

identify means for identifying one of the records in 
the storage means, the one of the records being identified 
15 by processing the likelihood value and the confidence 

value received by the input means, and each of the records 
stored in the storage means; and 

output means for outputting the indicator of the one 
of the records identified by the identifying means. 

20 

Thus, by outputting the indicator the user can 
determine the level of acceptability of the risk 
assessment. 

25 Preferably, the range of confidence values of the 

record identified by the identify means comprises the 
confidence value received by the input means, and the 
range of likelihood values of the record identified by the 
identify means either comprises the likelihood value 

3 0 received by the input means or is numerically closer to 

the likelihood value received by the input means than the 
range of likelihood values of any other of the records for 
which the range of confidence values comprise the 
confidence value received by the input means. 

35 

Preferably, the indicator in each of the records is a 
visual indicator each being a different colour or symbol. 
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Preferably/ the range of likelihood values in each of 
the records comprises a range of probability values. 

5 Preferably, the range of confidence values in each of 

the records comprises a range of integers. 

Preferably, the input means is configured to allow 
the user to enter and/or change the range of likelihood 
10 values and/or range of confidence values in each of the 
records . 

Preferably, the input means comprises a graphical 
user interface. 

15 

Preferably, the storage means comprises a computer 
storage medium. 

Preferably, the computer storage medium comprises a 
2 0 database. 



Preferably, the identify means comprises a suitably 
configured computer. 

25 Preferably, the output means comprises a graphical 

user interface. 

Alternatively, other forms of computer presentation 
devices can be used as the output means. For example, the 
3 0 output means could be a printer. 



According to a fourth aspect of the present 
invention, there is provided a method for assisting a user 
in determining a level of acceptability of an event 
35 occurring, the method comprising the steps of: 

receiving a likelihood value which represents a 
likelihood that the event will occur, and a confidence 
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value which, represents a level of confidence that the user 
has in the likelihood value; 

storing a plurality of records each of which includes 
an indicator representing a level of acceptability of the 
5 event occurring, a range of likelihood values, and a range 
of confidence values; 

identifying one of the records stored by the storing 
step, the one of the records being identified by 
processing the likelihood value and the confidence value 
10 received by the receiving step, and each of the records 
stored by the storing step; and 

outputting the indicator of the one of the records 
identified by the identifying step. 

15 Thus, by outputting the indicator the user can 

determine the level of acceptability of the risk 
assessment. 

Preferably, the range of confidence values of the 
2 0 record identified by the identifying step comprises the 
confidence value received by the receiving step, and the 
range of likelihood values of the record identified by the 
identifying step either comprises the likelihood value 
received by the input means or is numerically closer to 
25 the likelihood value received by the receiving step than 

the range of likelihood values of any other of the records 
for which the range of confidence values comprises the 
confidence value received by the input means. 

30 Preferably, the method further comprises the step of 

entering and/or changing the range of likelihood values 
and/or range of confidence values in each of the records. 

According to a fifth aspect of the present invention, 
35 there is provided an apparatus for determining a level of 
confidence in a risk assessment, the apparatus comprising: 
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obtaining means operable to obtain a weighting that 
is associated with at least one of a plurality of factors 
that can influence a reliability of the risk assessment; 
and 

5 comparing means operable to compare the weighting to 

a range of weightings which are associated with the 
factors in order to determine the level of confidence in 
the risk assessment. 

10 Preferably, the obtaining means is such that it uses 

the at least one of the factors to retrieve the weighting 
from a record of the weightings. 

Preferably, the obtaining means is operable to 
15 retrieve the weighting from the record by using a first 
index and a second index to retrieve the weighting from 
the record, wherein the first index corresponds to a first 
of the at least one of the factors, and the second index 
corresponds to a second and a third of the at least one of 
20 the factors. 

Preferably, the first of the at least one of the 
factors relates to an environmental parameter associated 
with the risk assessment, the second of the at least one 
25 of the factors relates to an extent of work performed to 
determine the risk assessment, and the third of the at 
least one of the factors relates to a level of acceptance 
associated with the amount of work performed and a 
technique used to perform the work. 

30 

Preferably, the record comprises a matrix that 
contains an entry for each of the weightings, and the 
first index and the second index correspond to an x, y 
coordinate for the entry. 

35 

According to a sixth aspect of the present invention, 
there is provided a method of determining a level of 
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confidence in a risk assessment/ the method comprising the 
steps of: 

obtaining a weighting that is associated with at 
least one of a plurality of factors that can influence a 
5 reliability of the risk assessment; and 

comparing the weighting to a range of weightings 
which are associated with the factors in order to 
determine the level of confidence in the risk assessment* 

10 Preferably, the step of obtaining the weighting 

comprises using the at least one of the factors to 
retrieve the weighting from a record of the weightings. 

Preferably, the step of using the at least one of the 
15 factors to retrieve the weighting comprises the step of 
using a first index and a second index to retrieve the 
weighting from the record, wherein the first index 
corresponds to a first of the at least one of the factors, 
and the second index corresponds to a second and a third 
20 of the at least one of the factors. 

Preferably, the first of the at least one of the 
factors relates to an environmental parameter associated 
with the risk assessment, the second of the at least one 
25 of the factors relates to an extent of work performed to 
determine the risk assessment, and the third of the at 
least one of the factors relates to a level of acceptance 
associated with the amount of work performed and a 
technique used to perform the work. 

30 

Preferably, the record comprises a matrix that 
contains an entry for each of the weightings, and the 
first index and the second index correspond to an x, y 
coordinate for the entry. 
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Notwithstanding any other embodiments which may fall 
within the scope of the present invention, an embodiment 
5 of the present invention will now be described, by way of 
example only, with reference to the accompanying figures, 
in which: 



Figure 1 illustrates an apparatus for assisting a 
10 user in determining a level of confidence in a risk 

assessment according to the embodiment of the present 
invention; 

Figure 2 illustrates an apparatus for assisting a 
15 user in determining the acceptability of an event 

occurring according to the embodiment of the present 
invention; 



Figure 3 illustrates a matrix stored in the storage 
20 means shown in figure 1; 

Figure 4 shows a table mapping sub- factors forming 
the environment factor stored in the storage means 
illustrated in figure 1; 

25 

Figure 5 shows a matrix used to derive the indexes 
shown in the table of figure 4; and 

Figure 6 shows information stored in the storage 
3 0 means illustrated in figure 2; and 

Figure 7 shows information related to factors that 
can be entered into the system of figure 1; and 



35 Figures 8 to 11 shows several images of the graphical 

user interface displayed by the system of figure 1. 
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AN EMBODIMENT OF THE INVENTION 

As can be seen in figure 1, the embodiment: of the 
apparatus 1 for assisting a user in determining a level of 
5 confidence in a risk assessment comprises: an input means 
3; storage means 5; processing means 7; and output means 
9. 

The apparatus 1 is in the form of a computer. The 
10 input means 3 is in the form of software which when run on 
the apparatus 1 effects display of a graphical user 
interface (GUI) on a visual display of the apparatus 1. 
The graphical user interface of the input means 3 is 
arranged to allow the user to enter a set of factors, each 
15 of which can effect a level of reliability in the risk 
assessment. 

The set of factors entered into the input means 3 
comprises: information about the environment of the risk 
20 assessment; information about the technique used to obtain 
the risk assessment; information about an extent to which 
the technique is used to obtain the risk assessment; and 
information about a level of acceptance of the technique 
and the extent to which the technique is used. 

25 

The information about the technique describes the 
technique employed to arrive at the risk assessment. For 
example, it describes the fact that the risk assessment 
was derived from either a personal assessment or by using 

30 a more formal technique such as statistical analysis to 
obtain the risk assessment. With regard to the extent to 
which the technique is used to obtain the risk assessment, 
this describes, for example, the number of samples on 
which the technique of statistical analysis was applied. 

35 The level of acceptance describes the number of people 
who accept the technique applied to the extent it is 
claimed to be applied. Finally, the information about the 
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environment describes the environment in which risk 
assessment is applicable. For instance, where the 
environment can be readily controlled (for example, a 
manufacturing environment) the environment is described as 
5 being relatively simple. In contrast, where the 

environment relates to the share market, in which there 
are many variables that can not be controlled or easily 
predicted, the environment is described as being highly 
complex. 

10 

Thus, the input means 3 (which as mentioned 
previously, displays a graphical user interface) allows 
the user to enter, for example, the fact that the risk 
assessment was calculated using statistical analysis and 
15 that the environment is highly complex 

The storage means 5 is in the form of a computer 
storage medium, and in particular is a database such as 
that available from Oracle. The storage means 5 stores a 

2 0 record containing a plurality of factors that can 

influence the reliability of the risk assessment. The 
storage means 5 also stores a plurality of weightings that 
are associated with the factors stored in the storage 
means 5. Essentially, each of the plurality of weightings 

25 represents a level of confidence in the risk assessment in 
relation to particular weightings. 



More specifically, the factors and weightings are 
arranged as a matrix 10 (see figure 3) . The weightings in 
3 0 the record are entries in the matrix 10, whilst the 

factors in the record act as an index 11 and 13 to the 
entries in the matrix 10. 

Each of the indexes 11 represent an environment in 
35 which the risk assessment was made, and is divided into a 
number of different levels. The symbols le - 4e of the 
matrix 10 in figure 3 are a convenient means for 
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representing each of the different levels of the 
environment. The mapping of the symbols to the different 
levels of the environment is shown in figure 4. The 
indexes 13 of the matrix 10 in figure 3 are derived from 
5 another matrix 14 which is shown in figure 5. The matrix 
14 also has indexes 15 and 17 , and entries which are the 
symbols (lb - 12b) used as the indexes 13 in the matrix 10 
of figure 3 . The indexes 15 represent the extent of work 
performed by the technique to derive the risk assessment. 

10 As can be seen in figure 5, the extent of the work 

performed is divided into a number of different levels, 
which include: basic, moderate, and extensive. As an 
example, if statistical analysis is used to derive the 
risk assessment and only a small number of samples are 

15 used, then the level of work performed is considered 

*\basic". However, if a large number of samples were used 
then the level of work performed would be considered as 
^extensive" . 



20 The indexes 17 represent a level of acceptance of the 

technique and extent to which the technique is used to 
calculate the risk assessment. As can be seen in figure 5, 
the indexes 17 are divided into a number of different 
levels, which comprise: single view; small group 

25 validation; large group validation; and widely accepted. 
Thus, by using the matrix 14 illustrated in figure 5 one 
can readily determine what the symbols 13 in figure 3 
represent. For example, the symbol 6b in the matrix 10 of 
figure 3 represents that extensive work was performed in 

30 obtaining the risk assessment, and that the technique 
employed to obtain the risk assessment is considered 
acceptable by a small group of people. 

The user of the apparatus 1 is required to enter into 
35 the input means 3 the following factors: the environment ; 
the level of acceptance of the technique and extent to 
which the technique is used to obtain the risk assessment, 
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and the extent of work performed by the technique. More 
specifically/ the user would enter the level of the 
factors applicable to the risk assessment. For example, if 
the environment was complex, the user would enter 
5 "complex" into the input means 3 . A summary of the various 
levels of the factors entered into the input mean 3 is 
provided in figure 7. 

Once the level of the factors have been entered into 
10 the input means 3, the processing means 7 proceeds to 

process the factors entered into the input means 3 and the 
matrix 10 in order to identify the indexes 11 and 13 which 
match the factors entered into the input means 3 . For 
example, if the factors entered into the input means 3 
15 where a "high-moderate" environment with "extensive" use 
of a technique which is accepted by a "large group", then 
the processing means 7 would identify the symbols "3e" and 
"9b" . The symbol "3e" being derived from the table shown 
in figure 4, whilst the symbol "9b" is obtained from the 
20 matrix 14 shown in figure 5. Once the processing means 7 
has determined the appropriate symbols, it proceeds to 
identify the equivalent indexes 11 and 13 in the matrix 10 
of figure 3 • 

25 It is noted that the processing means 7 is in the 

form of software that is executed by the apparatus 1. 

Once the processing means 7 has identified the 
appropriate indexes 11 and 13, the output means 9 proceeds 

30 to retrieve the weighting which is located at the 

intersection of the index 11 and 13 which have been 
identified by the processing means 7. In the case of the 
example given in the above paragraph, the intersection of 
3e and 9b is a weighting of "7". The retrieved weighting 

35 factor along with the range of weighting factors in the 
matrix 10 (that is, 2 - 10) is then output by the output 
means 9. By outputting the weighting "7" and the range "2 
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- 10" the user is able to put the weighting into context. 
In the case of the given example, one can see that "7" 
rates relatively highly of the range of ™2 - 10". Thus, 
assuming 10 represents a high level of confidence in the 
risk assessment and 2 represents a low level of confidence 
in the risk assessment, one can safely say that the 
weighting of rates relatively highly on the range of 

™2 - 10" . Consequently, a risk assessment of ™7" means 
that the user can have a reasonable level of confidence in 
the risk assessment. 

It is noted the output means 9 is in the form of 
software that runs on the apparatus 1, and which effects 
display of a graphical user interface on the visual 
15 display of the apparatus 1. 

In the present embodiment, the matrix 14 and the 
table illustrated in figure 4 are defined by a user of the 
apparatus 1. The input means 3 allows the user to define 
20 the matrix 14 and the table via the graphical user 

interface. Similarly, the entries in the matrix 10 are 
defined by the user via the graphical user interface of 
the input means 3 . 

In addition to assisting a user to determine a level 
of confidence in the risk assessment, the apparatus 1 also 
enables the user to determine a level of acceptability of 
an event occurring. In this regard, the apparatus 1 also 
comprises an identifying means 19. The input means 13 is 
such that it enables the user to enter a likelihood value 
which represents a likelihood that the event will occur, 
and a confidence value which represents a level of 
confidence that the user has in the likelihood value. 

35 The likelihood and confidence values entered into the 

input means 3 are in the form of a probability value (for 
example, 0.2 %) and integer value (for example 3), 



25 



30 



WO 2004/046976 



PCT/AU2003/001556 



-Ir- 
respectively. 

The storage means 5 is such that it stores a 
plurality of records, each of which comprises an indicator 
5 representing a level of acceptability of the event 

occurring. The record also comprises a range of likelihood 
values and a range of the confidence values. 

The indicators in the storage means 5 are visual 
10 indicators, each of which is a different colour or symbol. 
Where, for example, the indicator represents a very low 
level of acceptability, the relative indicator is the 
colour red, whilst a high level of acceptability is 
represented by the colour green. Where an indicator 
15 represents a medium level of acceptability, the indicator 
is orange in colour. 

The range of likelihood values in each record is 
essentially a different range of likelihood values. For 
2 0 example, a first record can have the range of < 0.5%, a 

second record can have the range 0.5% - 1.05% and a third 
record can have the record > 1.05%. The range of 
confidence values are ranges of confidence values (which 
are integers), such as 8 - 10, 5-10 and 1-10. 

25 

Figure 6 provides a conceptual diagram of the records 
that are stored in the storage means 5. With reference to 
figure 6, the indicators are marked 21, the ranges of 
likelihood values are marked as 23 and the ranges of 

30 confidence values are marked as 25. Once the likelihood 

value and confidence value have been received by the input 
means 3, the identify means 19 proceeds to process the 
received likelihood and confidence values and the ranges 
of likelihood values and the ranges of confidence values 

35 in each of the records in order to identify one of the 
records* The identifying means 19 is in the form of 
software that rims on the apparatus 1. 
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The range of confidence values of the record 
identified by the identifying means 19 comprises the 
confidence value received by the input means 3, and the 
5 range of likelihood values of the record identified by the 
identifying means 19 either comprises the likelihood value 
received by the input means 3 or is numerically closer to 
the likelihood value received by the input means 3 than 
the range of likelihood values of any other of the 
10 records. 



For example, where the input means 3 receives a 
likelihood value of 0.55% and a confidence value of 4, the 
identifying means 19 identifies the bottom record (of 
15 figure 6) ; that is, the record having a likelihood range 
of > 1.05% and a confidence range of 1 - 10. 

After the identifying means 19 has identified a 
record, the output means 9, which is in the form of 

20 software that effects a display of a graphical user 

interface on the monitor of the apparatus 1, retrieves the 
indicator from the identified record and displays it to 
the user via the graphical user interface. Thereby 
assisting the user is assessing the acceptability of the 

25 risk assessment. 

The input means 3 is also configured to allow the 
user to enter and/or change the range of values and/or 
range of confidence values in the records. This is 
30 provided for via the graphical user interface of the 

output means. The input means 3 also allow the number of 
indicators to be changed. 



With regard to figures 8 to 11, these figures 
35 represent the graphical user interfaces that enables the 
user to enter information (e.g. factors) into the system 
1, and display information to the user, (e.g. weightings) . 
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Those skilled in the art will appreciate that the 
invention described herein is susceptible to variations 
and modifications other than those specifically described. 
5 It should be understood that the invention includes all 
such variations and modifications which fall within the 
spirit and scope of the invention. 



